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ABSTRACT 

This thesis describes the formative evaluation of the Tactical Patrol Craft Trainer 
(TPCT) during the implementation stage. The TPCT is an interactive multi-media computer- 
based trainer designed to deliver a full fidelity and analog video training scenario to 
Prospective Commanding Officers (PCOs) of the Coastal Patrol Craft (PC) class ship. The 
system is designed to induce stress while enhancing autonomous decision making skills. The 
methodology involved six survey instruments, observation, and interviews with the trainees. 
The data show that the TPCT induces stress as intended, the user interface is appropriate, and 
the trainees perceived the training as valuable. Several improvements are noted in the course 
structure, user-interface, and system application. Recommendations are made for more 
scenarios, additional applications of the technology, and evaluation of training effectiveness 


after the final implementation of the system. 
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I. INTRODUCTION 
A. BACKGROUND 

i Simulation and Instructional Technology 

From the beginning of this century simulators have been a necessary tool for safer and 
more efficient flight training. The trainers in use during this period were mechanical mock- 
ups that simulated aircraft control. With advances in computer technology came advances 
in instructional technology. The U.S. Navy was experimenting with and studying instructional 
media and simulation as early as 1941 (Ellis, 1986). Distance learning, video 
teleconferencing, computer-based simulations, CD ROM courses, and the Internet are all 
advances in instructional technologies. Most of the technological advances are computer 
based including the use of multimedia computer video formats that integrate traditional video 
with computer power. 

We can do things with technology that are not possible without it. For instance, we 
can simulate flight for the novice; we can show how truly volatile a chemical is without 
exposing the student to risk; and we can put the trainee in simulated combat situations that 
were not possible before. Advances in instructional technology have provided us with a 
means to enhance the training environment without putting the trainee in danger or resulting 
in the loss of valuable equipment. The development of “safe, novel training techniques, which 
could be achieved through simulation and mass training media” was the charter behind the 
establishment of the Navy’s Special Devices Center in 194] (Dreves, 1971). Studies as early 
as the 1940s proved that simulators could effectively reduce actual flight training time (Ellis, 


1986). Flight training simulators led to developments in other simulators that provided 


effective and safe training alternatives including nuclear reactor trainers and surface ship 
trainers. 

Ze Tactical Patrol Craft Trainer (TPCT) Background 

The TPCT was developed to fill a void in the training pipeline of U.S. Navy Officers 
who would take command of a Coastal Patrol (PC) class ship. High stress autonomous 
decision making skills are not part of the normal training pipeline or experience level of these 
officers. The need was apparent for a system that would reinforce selected training points and 
enhance the decision making capability, under stress, of the prospective Commanding Officer 
(CO). Commander Naval Special Warfare Command’ (COMNAVSPECWARCOM) needed 
a training system that would stress the student in a realistic environment, enhance advanced 
decision making skills, and emphasize training elements of concern to the Coastal Patrol Boat 
(PC) community. 

The Integrated Damage Control Training Technology (IDCTT), an interactive 
multimedia system developed for training Damage Control Assistants (DCAs) on Naval Ships, 
provided “levels of reality and stress that have not been experienced in damage control 
training before” (Fuller, 1993). Developers decided that the IDCTT could be adapted to 
create a system capable of producing the desired training environment for PC Commanding 
Officers. The IDCTT is discussed in further detail in the literature review section of this 
thesis. From a statement of requirements for further training and the use of current 


technology, the TPCT was developed. 


'The PC class ship is directly under the control of COMNAVSPECWARCOM. 
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Systems Integration & Research, Inc. (SIR) was awarded the contract by 
COMNAVSPECWARCOM to assemble the full fidelity computer hardware components, 
develop and program’ the initial TPCT scenario, produce the analog video segments for 
incorporation into the scenario, and program the interface for the Integrated Bridge System 
(IBS).’ SIR was also tasked with installing the system, conducting initial system testing and 
evaluation, and implementing the system. This thesis was conducted with the support of 
COMNAVSPECWARCOM dung the initial installation phase. 

B. AREA OF RESEARCH 

The purpose of this research is to evaluate the TPCT during its installation phase. 
This research is the beta testing of an interactive multimedia computer training system prior 
to general use. Beta testing is the pre-implementation evaluation of a computer system to 
determine if hardware or software improvements are needed prior to release to the customer. 
This formative evaluation analyzes the training system while test subjects use it before final 
implementation. This evaluation is an important part of computer system development. 
Goldstein (1993) indicates that it is essential for developers to determine, through formative 
evaluation, if improvements are needed before a training system is put in operation. 

c: RESEARCH QUESTIONS 
The primary research objective is to conduct formative evaluation of the trainer and 


make recommendations for implementation. Specific research questions are: 


* The programing was subcontracted to Tekamah, Inc. 


> The IBS is a navigation and control system developed by Sperry Marine, Inc. that 
is in use on the PC class ship. 


1. What system changes need to be achieved prior to implementation? 

a. What changes need to be made in the Instructor's Guide? 

b. What changes need to be made in the Student's Guide? 

c. What software problems need to be corrected? 

d. What hardware problems need to be corrected? 

e. What additional instruction needs to be conducted prior to using the trainer? 
2. What initial indications of system effectiveness are produced during beta testing? 


3. What variables need to be collected to create a database that will allow future analysis 
of training performance? 


4. During initial beta testing, what variables are significant factors in the successful 
completion of the trainer? 


D. SCOPE OF THESIS 

The primary thrust of this thesis is a formative examination of the TPCT in beta testing 
and recommendations for implementation. The thesis identifies training system improvement 
needs, initial indications of system effectiveness, and initial factors of successful trainer 
completion. This study is not a comparative analysis nor does it address cost effectiveness. 
The TPCT is still in the installation phase and the analysis will be an integral element of the 
beta testing. The analysis does not measure the ultimate effectiveness of the TPCT. 
E. ORGANIZATION OF THESIS 

Following the introduction, this thesis is organized into five chapters. First the 
literature review discusses those areas pertinent to the thesis. Works are reviewed that detail 
beta testing, simulation (including the IDCTT), training and evaluation, and those works used 


to evaluate decision making. The next chapter is a detailed TPCT description. This chapter 


familiarizes the reader with the basic TPCT concept, the physical components (system 
hardware), and the software (including basic scenario description). This chapter provides the 
reader with fundamentals to understanding the research that follows. 

The research chapters begin by descnbing the methodology. The methodology that 
was planned in the research design phase is contrasted with the actual methodology used. 
The shift in methodology was an important factor in the collection of data. Next is a 
presentation of the data and analyses followed by conclusions and recommendations. The last 


section also includes current status of the recommendations’. 


*Recommendations were submitted to COMNAVSPECWARCOM during the 
testing phase. 





Hf. LITERATURE REVIEW 


The literature reviewed pertaining to this research is divided into three sections. The 
first section looks at literature in the area of training evaluation. Second, beta testing and 
computer system evaluation literature is reviewed. Third, is a review of the literature on 
technology including simulation and its use in training. This section includes studies that look 
at the Tactical Patrol Craft Trainer (TPCT) predecessor, Integrated Damage Control Training 
Technology (IDCTT), and the literature on decision making and the effects of stress is 
reviewed. 

A. TRAINING EVALUATION 

Evaluation is an integral part of a training system. It is not a stand-alone element. It 
must begin at inception and continue through the life of the system to provide feedback for 
system improvement and effectiveness. Goldstein (1993) describes a systems approach to 
training that includes needs assessment, development and training, evaluation, and validation 
in a looped feedback system. This approach would seem to indicate that evaluation has one 
definitive and specific place in the training system. But, further exploration of each system 
component makes it apparent that each phase contains its own element of evaluation that 
feeds back into the system (Goldstein, 1993). 

The evaluation phase in this model deals with the summative and formative evaluation 
of a training system. Summative evaluation primarily looks at expected outcomes of a 
completed system or whether the system is more effective than an alternative system. 


Summative evaluation also includes cost-benefit analysis. Formative evaluation tests review 


a training system during its design and implementation to ensure that basic concepts work. 
The difference between summative and formative evaluation, and the importance of both as 
separate instruments, is important to the evaluation conducted. As noted by Goldstein, 
“[mJany research problems result from one-shot evaluation studies that attempt to combine 
formative and summative evaluations” (Goldstein, 1993). To have an effective system, 
summative evaluation must follow formative evaluation in the process of training system 
development (Goldstein, 1993). This thesis does not attempt summative evaluation and is 
concerned only with the formative evaluation in the implementation phase. 
B. COMPUTER SYSTEM EVALUATION 

Computer system evaluation can take on many forms and can occur at various points 
in the system’s life cycle. In the beginning, hardware tests may include mechanical stress and 
electronic congruity testing. The software may undergo programmer logic checks that 
determine if the code operates as expected. Toward this end, there are software tools 
designed to test a computer system. For example, Rose (1996) prepared an extensive article 
about the use of automated testing tools in the quality assurance of computer systems. Her 
assertion is that the use of automated test tools is a necessary element in the testing cycle to 
reduce the time-intensive process of manual testing. But in the article she also emphasizes 
that the formative evaluation, what she calls beta testing, involving the end user, can not be 
completely replaced with automated testing. In fact, she indicates that the formative 
evaluation of the user interface is essential to the successful performance of the system. 

A procedure for evaluating the user interface performance of computer-based 


multimedia systems for training 1s described by Reeves and Harmon (1994). The procedure, 


“User Interface Dimensions,” is based on elements that evaluate usability of the system. They 
present a thesis in which the value of a computer-based training system can be evaluated 
based on whether “users have a meaningful and purposeful experience” while using the 
system. The procedure evaluates user responses in areas “such as ease of use, media 
integration, and cognitive load” (Reeves & Harmon, 1994). They set up ten dimensions that 
are presented without being numerically grounded. The dimensions are either negative, 
positive, or somewhere in between. They do, however, recommend the eventual grounding 
of these dimensions on a ten-point rating scale. 

Reeves and Harmon (1994) admit that the dimensions have not been extensively tested 
or completely validated and “should be subjected to mgorous expert review.” But, they do 
describe a very detailed use of the procedure to evaluate a computer-based multimedia 
educational program. The test indicated that users generally had a hard time operating the 
system and because of this, the system’s utility as a training device was questioned. While 
this procedure may tell us something about the usability of the system, it still lacks some 
elements that are needed to completely evaluate a training system. The dimensions are not 
instruments of summative evaluation and do not evaluate system effectiveness in a 
quantitative manner or in comparison with other methods of teaching the same subject. 

This thesis is a formative evaluation and the “User Interface Dimensions” is used to 
indicate areas that need improvement. The dimensions will help determine an initial measure 


of effectiveness by evaluating the usability of the system. 


C. SEMULATIONS AND INSTRUCTIONAL TECHNOLOGY 

The U.S. military’s use of simulation as a training device is extensively chronicled in 
a series of articles edited by Ellis (1986). Included in this series are articles on needs 
assessment, training system design, evaluation, and validation. All the articles emphasize 
computer-based or simulator-based training that, when combined, resemble Goldstein’s 
(1993) approach to training. Although most of the evaluation procedures are summative, the 
need for separate formative evaluation prior to implementation is emphasized. A review of 
training devices and computer-based training in the military is presented that provides a 
history of military uses, deficiencies in training devices, effective uses for computer-based 
training, and other issues relevant to the development of these systems. The following 
sections are based on findings by Ellis (1986) and were used to generate assumptions for this 
thesis. 

1. Ease-of-use 

Ellis (1986) found that ease of use and properly implemented training devices are 
essential to the effectiveness of training systems. This finding agrees with the assumptions 
behind the development of Reeves and Harmon’s “User Interface Dimensions” and the use 
of them in this thesis. Johnson (1994) used the dimensions to conduct a formative evaluation 
of the IDCTT during its implementation. The student responses to the “User Interface 
Dimensions” indicated that the IDCTT had a very appropriate user-system interface. Johnson 
found that the students were receptive to the training and found it easy to use. Baumann, et 
al. (1996a) conducted a program review of the IDCTT that validated Johnson’s (1994) 


findings. In the program review the researchers also used the “User Interface Dimensions” 
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and determined that the IDCTT was perceived as easy to use. During implementation of the 
TPCT, this research also used the “User Interface Dimensions” to evaluate the system’s 
usability. The assumption for this research was that problems with usability would inhibit the 
training system’s effectiveness. 

ps Touchscreen Monitors and Graphics 

Although Johnson’s (1994) IDCTT results were generally favorable, there were some 
needed improvements noted. The touchscreen monitor was difficult for students to operate 
and was evaluated as a major problem in the system. Ellis (1986) found that powerful 
computer graphics and touch screens are important and desirable parts of a computer-based 
training system. Because of Ellis’ assertion and Johnson’s finding, the touchscreen monitor 
for the TPCT was closely evaluated during this research. Touch sensitivity must be set to a 
usable level and the graphics for a touchscreen monitor must be integrated in a usable and 
understandable way. If the buttons are too small, the user will have a hard time selecting the 
appropnate response item. Ifthe graphics are hard to see or not clear, the user will be unable 
to find the appropriate response item. So, not only do the graphics provide important 
“attention value” (Wetzel, et al. 1994), but they are also an essential element in the 
interactivity of the system. 

Fletcher’ s (1990) research indicates high utility for interactive video training. Fletcher 
(1990) reviewed 28 studies and concluded that “higher levels of interactivity were associated 
with higher levels of achievement.” Fletcher (1990) also came to the conclusion that 
interactive video-based training can be more efficient and more effective than conventional 


training methods. With increases in technology comes the ability to increase the graphic 


I] 


capability of a training system and this should enhance realism in an interactive video-based 
traiming system. This thesis is concerned with the trainees’ perceptions of system realism and 
how this affects the interactive capability of the system. Other technological advances 
including more computer power, voice recognition, and high levels of fidelity enone 
computer-based systems to increase levels of realism (Ellis, 1986). 

a Fidelity 

Ellis (1986) determined that fidelity is important to the training system, but the level 
of fidelity must be consistent with the training requirements. Fidelity is also emphasized as 
a major component in simulation training technology by Wetzel, et al. (1994). But, they also 
caution that interactive training systems based on video be developed with a critical review 
of the need for fidelity. Functional fidelity (the amount a system acts like the real thing) is 
considered an important element in the effectiveness of a training system. But, the question 
is raised as to whether simulations must also look like the real thing (physical fidelity) to be 
effective. Studies cited in Wetzel, et al. (1994) indicate that extremely low or extremely high 
levels of fidelity can actually impede the learning process for the beginning learner’. But, 
“Tals students become increasingly sophisticated, better learning is achieved with increasingly 
higher fidelity instruction that, these students are better prepared to process” (Wetzel, et al. 
1994). The assumption in this thesis is that the trainee is not a “beginning learner” and that 


evaluation of both functional and physical fidelity will indicate if the system is productive. 


°The definition of “beginning learner’ as well as definitions of high and low levels 
of fidelity are not expanded on by Wetzel, et al. (1994). 
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4. Trainee Perceptions 

The IDCTT scenario was considered realistic and the high fidelity of the system was 
responsible for student perception of the trainer as an effective training method (Johnson, 
1994). According to Ellis (1986), adverse perceptions of the training system by the trainee 
will result in poor training outcomes. Johnson found that the students were receptive to the 
training and concluded that this was a factor in the system’s effectiveness. Baumann, et al. 
(1996a) concurred with Johnson’s conclusion and noted that the training was perceived as 
valuable. This thesis looks at pre-training perceptions of simulation training and follows up 
with the trainees’ perceptions of the TPCT in various areas. If the trainees perceive the 
TPCT as an effective training method, the indication is that there is less possibility of the 
system producing poor outcomes. Negative perceptions by the trainees led this research to 
look for causes and possible corrections. Of particular interest is the trainee’s perception of 
stress induced by the trainer. 

a Stress and Decision Making 

The TPCT was designed as a high-fidelity fast-paced multimedia system that creates 
a stressful environment in which decisions are made. Decision making under stress 1s the “job 
response” that the training system is intended to enhance. Computer-based training can be 
an effective means to “fine tune, integrate, and strengthen job responses” (Ellis, 1986). But, 
“Tolne of the most difficult topics to quantify in training is the amount of stress in an 
environment and its effect on an individual’s ability to make decisions” (Fuller, 1993). Stress 
induced by the system is intended to acclimate the trainee to the decision making process in 


a stressful environment. The acclimation to stressful environments combined with expert 
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knowledge and confidence will reduce the perceived level of stress and enhance performance 
(Heslegrave & Colvin, 1996). But, the challenge in this research was how to determine if the 
system was producing appropriate levels of stress. 

The “NASA-TLX” was developed to assess the relative importance of six factors in 
determining how much stress or pressure was experienced during a task or exercise (Hart & 
Staveland, 1988). The “TLX” is a self-report mechanism that measures perceived amounts 
of: 1) Mental Demand, 2) Physical Demand, 3) Temporal Demand, 4) Performance Demand, 
5) Effort Demand, and 6) Frustration. These factors were validated as effective in identifying 
variations in stress and diagnosing sources of stress within a task (Hart & Staveland, 1988). 

The ability of the “TLX” to identify sources of stress within a task or system was an 
important factor in a review by Hill (1992). Four subjective rating measures of stress were 
compared by Hill, and the “TLX” was determined to provide the best source for determining 
source of stress in a task or system (1992). Because of the validation of the “TLX,” Hill’s 
assertion of its usefulness, and use of the “TLX” by Johnson (1994) and Baumann (1996a), 
it was used in this research to determine the stresses that are created by the TPCT system. 
Evaluating the stress induced by the TPCT system will determine if the system is operating 
as intended. But, it does not address the question of the effectiveness of these stressors. 
Effectiveness will need to be evaluated in future summative evaluation. 

6. Effectiveness and Efficiency 

Computer-based training provides a means to effectively train otherwise unsafe or 
economically impossible tasks (Ellis, 1986). This research is not looking at outcome-based 


effectiveness or cost savings. But, the literature review provides clues as to where a system 
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may need evaluation during implementation to enhance the possibility of effectiveness or 
prevent costly redesign of the system after implementation. Offering words of caution, 
DeBloois (1988) indicates that development of an effective interactive multimedia training 
system is a complicated and time-consuming project. But through the review of 30 studies, 
DeBloois (1988) found a variety of interactive multimedia training systems were effective and 
efficient. Of particular interest in this thesis are the results of the “VISTA 1982” study that 
used videodisc training for leadership skills and the “IBM Management Training 1986” study 
that used videodisc for management training. In each of these studies DeBlooise (1988) 
reported that the videodisc training was significantly more effective compared to more 
traditional forms of training, including role playing. Because the TPCT is designed to 
enhance leadership and management skills (1.e., decision making) the results of these studies 
indicate that the use of videodisc training is a reasonable method to use. 

Johnson’s (1994) summative evaluation compared student performance in the IDCTT 
trainer with performance in a “mock-up” trainer system that was already in place. Comparing 
the performances of students in the study, Johnson concluded that the IDCTT was more 
effective than the system already in use. Baumann, et al. (1996a) conducted a program review 
of the IDCTT that validated Johnson’s (1994) findings. In the program review the IDCTT 
was determined to be mentally demanding and to produce high levels of stress and frustration 
(Baumann, et al. 1996a). These researchers also concluded that the IDCTT was more 
effective than the current “mock-up” trainer, that multiple scenarios are needed, and that the 
system provided valuable training. Since the TPCT is based on the IDCTT, similar results 


were expected and similar measurement instruments could be used. 


Is 


A cost benefit analysis was conducted of the IDCTT system for Naval Sea System 
Command (NAVSEA, 1996). The draft report concluded that the IDCTT was cost effective 
relative to current training methods, and that there was future utility for the system. Included 
in this analysis was an endorsement for future scenarios addressing the recommendations of 
Johnson (1994) and Baumann, et al. (1996a). The need for scenario variety in the IDCTT 
prompted a goal for this research to specifically look for recommendations for new and varied 
scenarios for the TPCT. 

The Fuller Decision Index (FDI), a quantitative measure of proficiency for the IDCTT, 
was used to support claims of effectiveness. The FDI “is a decision making quality 
measurement system that was developed with the goal of objectively and statistically deriving 
expert performance standards for the IDCTT” (Fuller, 1993). Although the FDI was not used 
in this research, Fuller’s review of decision making theory and stress provided background. 
The IDCTT is the predecessor to the TPCT, and a review of the literature on the IDCTT 
provided direction for this thesis. 

Johnson’s (1994) formative evaluation of the IDCTT laid the groundwork for this 
thesis. Johnson evaluated the IDCTT with the “User Interface Dimensions,” the “NASA 
Task Load Index” (“TLX’), and survey responses from students. Students were more 
responsive to the interactive multimedia training and it “promoted greater learning, produced 
significantly more stress, and stimulated students” (Johnson, 1994). The student responses 
to the “User Interface Dimensions” indicated that the IDCTT had a very appropniate user- 
system interface, and the responses to the “TLX”’ indicated perceived high levels of temporal 


demand on the students. The fast pace of the program caused minor problems that were 
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remedied by students conducting tnal runs on the system. The students indicated that the 
system was limited and there was a need for more scenarios and different ship classes. The 
preceding items are not inclusive of Johnson’s findings, but they represent some of the major 
problems that were kept in mind when developing the methodology for this thesis. 

The literature reviewed in this section provided the foundation from which this research 
was designed. Before the methodology and use of the tools mentioned in the literature review 


can be discussed, the following chapter presents a complete description of the TPCT system. 


le 


= 





Wl. TACTICAL PATROL CRAFT TRAINER DESCRIPTION 


The Tactical Patrol Craft Trainer (TPCT) is an interactive multi-media computer-based 
trainer designed to deliver a realistic full fidelity and analog video training scenario. The 
trainee sits (but could stand) in front of two 20" touch screen computer monitors that create 
a semi-virtual environment simulating the actions of a PC commanding officer. Unlike a total 
virtual environment, the TPCT trainee is subject to the external environment. The physical 
presence of the instructor, room temperature, and outside noises could potentially distract the 
trainee during the scenario. The system currently has one scenario and the intended trainee 
is a prospective PC commanding officer, normally a Navy lieutenant, en route to PC 
command. The TPCT instructor is not intended to interact with the trainee during the 
scenario but is available if problems arise. This chapter introduces the reader to the major 
components of the TPCT system. The three basic components are discussed: 1) hardware, 


2) mission scenario, and 3) software. 
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A. TPCT HARDWARE COMPONENTS 

The hardware ccraponents of the TPCT, as illustrated in Figure 1, consist of an 
instructor’s TPCT rack that contains the major components to run the simulation and two 
monitors for use by the trainee. The TPCT rack is a multi-component system that runs the 
simulation. It contains four networked Central Processing Units (CPUs), a laser disk player, 


keyboard and mouse, and a 14-inch monitor. Before any training starts, all of the system 
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TPCT hardware components. Source: Instructor Guide. 
components must be initialized in a precise process that must follow a strict order. The 
initialization process is detailed in the instructor’s operating guide and discussed later in this 


chapter. Together, these components control all aspects of the simulation and allow the 


instructor to aid and monitor the trainee. 
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1. TPCT Trainer Monitor 

The over-the-counter (OTC) monitor, shown in Figure 2, is used as the communication 
link between the trainee and the scenario. During testing, this monitor is positioned to the left 
of the trainee. This 20-inch polished-finish® monitor uses touch screen technology. A touch 
panel on the screen is used for input to the system by the trainee. A keyboard and mouse are 
also available but were not used during these test because they are currently considered back- 


ups to the touch screen technology. The trainee views video segments relevant to the 
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Figural 2 TPCT Trainer Meiiians view during scenario. Source: eae Guide. 





°Polished-finish refers to the type of glass screen on the monitor. Polished-finish 
has a glossy look to it while an etched finish screen has a non-glare appearance. 
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scenario in the window on the left portion this screen. All of the video is from the viewpoint 
of the trainee, as if the trainee were looking at the action. When there is no relevant video 
segment, the area is black. Mission and ship’s status information is available either directly 
on the screen or through opening a window by touching a screen button. 

Status information includes such features as depth of the water, material condition, 
course, and speed. The trainee can view most of the information without opening another 
window. The status board and reports sections contain information that is not adjustable by 
the trainee. All other areas and functions on the screen can be altered by the student by 
pressing a button activating an on-screen menu of choices related to the desired action. 


Figure 3 illustrates the procedure for lighting off or securing the ship’s radar. Ifthe trainee 
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wants to secure the radar system, he/she would press the radar button, press “Secure” and 
then press “OK.” The action is not complete until OK or CANCEL is touched and no other 
action can be taken by the trainee while this window is open. Use of the on-screen menus is 
equivalent to issuing an order. The ship will respond to the request even if it leads the ship 
into harm’s way. Decisions are made by either ordering an action or by ignoring a request 
for action. At times, the video segment may recommend a course of action and the trainee 
can simply ignore the request. This is equivalent to a negative response. If the trainee elects 
to order the action, it is done through a series of on screen buttons. 

a Situation Display Monitor 

This is an OTC 20-inch monitor with an etched finish and touch screen capability. The 


Situation Display (see Figure 4) shows the output from the Integrated Bridge System (IBS), 
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Figure 4. IBS screen from Situation Display. Source: Student Guide. 


and it provides information to the trainee regarding the ship’s navigation. The only inputs 
from the student to this system are requests for information, adjusting views (zooming in or 
out), or selecting plans entered previously by the TPCT system for voyage management. The 
IBS uses information from the TPCT system to indicate the ship’s movement through the 
water. The TPCT’s IBS display is the same that is found onboard the PC class ship. During 
operation of the scenario, the grid area contains an area map representation, a visual 
representation of the ship, and the ship’s movement. This screen is positioned to the night of 
the trainee during testing. 

3. TPCT Rack 14-inch Monitor 

The 14-inch instructor’s monitor in the TPCT rack allows the instructor to follow the 
trainee through the scenario. The instructor can watch reproductions of either the Situation 
Display Monitor or the Trainer Monitor and can shift between the two during the training 
session. The instructor sees what the trainee sees but also sees information not available to 
the trainee. For example, the instructor can observe the trainee’s progress into a decision 
point area, indicated only on this screen and monitor progress without intruding on the 
physical space of the trainee. The monitor is also used during system initialization to conduct 
system analysis and testing. It can be used to monitor the actions of any of the four CPUs. 

4. Laser Disk Player 

The laser disk player, a generic Over the Counter (OTC) piece of equipment, is used 
to feed the system with analog video produced to enhance realism in the scenario. The 
current scenario contains video segments that appear only during appropriate times within the 


scenario. There is not a continuous video feed, and the laser disk is accessed only according 
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to scenario programing requirements. The video segments were professionally produced re- 
creations of events filmed onboard a PC class ship using a combination of actors and Navy 
members. 

>: Rack Mounted Keyboard and Mouse 

A standard OTC personal computer keyboard and mouse device are provided for 
instructor input into the system. The mouse allows for standard click and drag computer 
Operations. 

6. TPCT Trainer CPU 

The trainer is an OTC Pentium®-based CPU running a Windows® NT operating system. 
The TPCT trainer scenario software is run on this CPU. This unit also contains software that 
directs the actions of all CPUs and the laser disk player in a coordinated fashion as required 
by the scenario software. 

a. Situation Display CPU 

This CPU controls the Situation Display Monitor. It is a Pentium*-based CPU 
running an OS2 operating system provided by Sperry, Inc. The Situation Display Monitor 
is the useable and visual output of the IBS. The IBS also uses the next two CPUs. 

8. Simulation-Vision Technology CPU 

This Simulation-Vision Technology (Sim-VT) CPU interacts with all other CPUs and 
the laser disk to control the video throughout the simulation. This CPU also provides the IBS 
navigation data to the Voyage Management System-Vision Technology (VMS-VT). It is a 


Pentium®-based CPU running an OS2 operating system provided by Sperry, Inc. 
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9. Target Correlator CPU 

This CPU runs the Target Correlator Monitor (TCMON) and the Position Filter 
Module. This is the remaining component of the IBS using a Pentium®-based CPU running 
an OS2 operating system provided by Sperry, Inc. 

B. MISSION SCENARIO 

The TCPT scenario simulates a U.S. Navy Sea, Air, Land (SEAL) team insertion 
mission using the PC class ship as the insertion platform. The scenario begins with a mission 
brief to the trainee who is acting as the Commanding Officer of a PC. The trainee 1s provided 
with a computerized mission brief package that can be read prior to mission start and can be 
referred to during the mission. The trainee is then briefed in a video segment that simulates 
the actual brief that might occur by an operational commander prior to a mission. Both briefs 
contain mission plans, weather reports, enemy movement and capability reports, Standard 
Operating Procedures (SOPs), and other mission-critical information. After the mission brief, 
the system pauses until the trainee indicates readiness to start by pushing the “Start Mission” 
button on the TPCT trainer monitor. 

The mission begins with video simulating walking aboard the ship while the crew is 
making preparations to get underway. From this point forward, the trainee is required to 
make decisions based on computer-generated situations to complete the mission. The trainee 
must make the decision to get underway and order it by pressing the correct button on the 
Trainer Monitor. All of the actual actions required by ship’s company to get underway are 
then carried out correctly without further monitoring by the trainee. To complete the mission, 


the trainee must exit the friendly harbor, transit to enemy waters, insert a SEAL team in an 
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enemy harbor, stand off while the team conducts a mission, come back in and pick up the 
team, and exit enemy waters. The SEAL mission ashore is not a part of the training scenario. 
The trainee is expected to already have the knowledge base required to conduct this mission. 
But, this seemingly simple series of events becomes complicated rather quickly. 

The mission scenario is conducted under strict transit time requirements while dealing 
with a variety of problems and decisions along the way. The problems and their solutions are 
known as “Decision Points” or “Teaching Points.” The TPCT focuses the trainee’s attention 
on an abundance of information requiring timely response to the many significant decision 
points associated with a complex scenario. Depending on the nature of the response, the 
scenario either shifts toward resolution or degradation that can lead to prematurely ending 
the scenario. Some events, video segments, and sound bites are included in the scenario to 
create a fast-paced stressful environment under which the trainee must make important 
decisions. The scenario ends as either a mission success from the correct treatment of 
Decision Points or discussing with the instructor the appropriate decisions to emphasize a 
Teaching Point. 

An incorrect response to one of the Decision Points in the hour-long simulation’ will 
result in ending the program either through some direct catastrophic situation or inability to 
meet the transit time requirements. This is not considered a failure, rather it becomes a 


Teaching Point that can be discussed between the instructor and trainee. The trainee can then 


’The mission in this scenario would take over 6 hours to complete. The simulation 
is shortened through the use of “warping,” which advances the trainee and ship’s position 
in the mission at an accelerated rate. During this time no actions are taken by the trainee. 
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restart the mission from the beginning and again attempt mission completion. The scenario 
used in this evaluation was not alterable, and there was no way ‘'r the trainee to start where 
the last attempt ended. 

C. TPCT SOFTWARE COMPONENTS 

Software in a multimedia system refers to the programming code that causes the system 
to function. This thesis does not evaluate the programming except in reference to how well 
it operates. The programming code, per se, is not evaluated. Flaws in the code were noted 
if a failure occurred during operation of the system. Each CPU in the system contains its own 
set of code. The three CPUs that operate the IBS were pre-programmed by Sperry, Inc. and 
contained no unique programming. The programming for the TPCT Trainer CPU was new 
and contains the innovative networked link to the IBS system. This CPU facilitates the multi 
media interactive service. 

Integral to the software is the documentation for the system including the operating 
guide, the instructor guide, and the student guide. These documents were evaluated as part 
of this thesis. They were draft documents that needed testing and revision. The operating 
guide served to provide instructions to the instructor for operating the system, primarily for 
starting the CPUs and securing them after training. The instructor guide gave detailed 
scenario and training information with procedures for conducting the traimng. The student 
guide provided the student with a system overview and information on how to operate the 


system. All of these documents were draft copies. 
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IV. METHODOLOGY 


This chapter describes the methodology used in this research. The methodology 
evolved as a result of pre-testing the Tactical Patrol Craft Trainer (TPCT). An explanation 
of this evolution is provided. Detailed in this chapter are the subjects, data collection 
instruments, and procedures used to conduct the beta testing. 

A. SUBJECTS 

The proposed methodology was originally designed to compare the performance of 
approximately 10 Subject Matter Experts (SMEs) with two different groups of trainees: 1) 
the four current Patrol Craft (PC) Commanding Officers in San Diego, and 2) about 30 Navy 
Lieutenants*. The subject matter experts are Naval officers who have had command at sea 
and thus could establish a baseline for evaluating the performance of trainees on the system. 
The assumption was that the SMEs would complete the TPCT scenario in a specific number 
of runs and this could be compared to the number of runs required to complete the scenario 
by the trainees. This proposed comparison became problematic early in the research. 

The availability of test trainees, especially SMEs, was severely limited because of work 
schedules of potential subjects. There were no SMEs available during the period of testing 
for this research. An even more limiting factor, discovered in pre-testing, was the inability 


of the trainees to complete a successful scenario run in a 3-hour time frame. 


*These officers are representative of officers who might use the TPCT en route to 
command of a PC class ship. 
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Two test subjects, from the same pool of available waterfront personnel as the actual 
test subjects were pre-tested on the TPCT to collect pilot data. The pre-testing allowed the 
researcher to evaluate the time required to conduct each training session. Because the TPCT 
software requires the trainee to start from the beginning of the scenario after missing a critical 
decision point, it increased the testing time beyond the expected 3 hours and beyond the time 
most subjects could provide. Trainees were arranged by SPECWARCOM and were made 
available for a 3-hour training session. Another difficulty was the time required for trainees 
to complete the survey, review the student guide, and learn the basics of the Integrated Bridge 
System (IBS). Almost 1 hour of the allotted time was used to complete these aspects of the 
testing. Trainees who had not completed the PC Commanding Officer training pipeline were 
not familiar with the IBS prior to using the TPCT’. 

Navy commands in the area were contacted and asked for volunteers. Volunteers were 
also solicited from within SPECWARCOM. In the end, 18 test subjects participated 1n this 
study from various commands in the San Diego, CA Naval District. Seventeen were U. S. 
Navy officers and one was a senior enlisted member from a PC class ship with Officer Of the 
Deck (OOD) qualification’®. Of the 17 officers, 11 were Lieutenants, two were Warrant 
Officers, two were Lieutenant Commanders, one was an Ensign, and one was a Commander. 


The mean age of the group was 31 years old , the oldest was 45, and the youngest was 24. 


The testing also revealed that IBS would not be known to the trainees for whom 
the TPCT was designed because IBS is not taught in the PC commanding Officer training 
pipeline. 


‘OOD qualification is a bridge watch station that would require the member, in 
the absence of the CO, to make some of the same decisions the CO would make. 
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The education background, commissioning source, and ship experience demographics of the 


test group are shown in Table 1. 


Table 1. TPCT trainee demographics. 


Trainee Demographic Information 


Total Percent of Group* 


Education 
BS Engineering 17% 
BS Non-Engineering 61% 
BA 6% 
MS Engineering 0% 
MS Non-Engineering 0% 
MA 6% 
Ph.D. 0% 
OTHER 0% 


Commissioning Source 
USNA 39% 


NROTC 33% 

OCS 6% 

ECP 6% 

Seaman to Admiral 0% 
OTHER 0% 


Ship Types Served in 
CRUDES 67% 
AMPHIB 22% 
CV 0% 
AUX 11% 
PC 33% 
MCM 11% 
OTHER 0% 


At Sea Experience 
COMMAND 
XO 


Department Head Jobs 
Operations 

Deck 

Air 

Engineering 

Combat Systems 
Weapons 

Other 


Division Officer Jobs 
Engineenng 

Deck 

Air 

Operations 

Weapons 

Combat Systems 
Other 


* All percentages do not total 100. Trainees may have more than one degree and | 
could have experience in more than one ship or job class. Education and 
Commission Source also have missing data. This figure represents only the 
percentage of the test subjects with that particular demographic. 
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The instructor during this study was a civilian contractor. He 1s a retired U. S. Navy 
Commander with OOD qualification and experience in the development of the PC class ship 
and the development of the TPCT. The instructor 1s also qualified as a Navy Master 
Instructor and provided invaluable insight into the training process that resulted in a more 
comprehensive review of the instructor’s guide. 

B. DATA COLLECTION INSTRUMENTS 

A TPCT training data package was developed for this research. The complete 
package, which consists of pre-and post-training survey questions, interview questions, and 
instructions, is included in this thesis. A group of 10 Naval Postgraduate students evaluated 
the training data package for clarity and understanding prior to use in this research. Field 
notes taken during the testing and a report generated by each TPCT run were used in this 
research to compare each trainee’s survey responses. Additionally the items discussed below, 
which were contained in the TPCT training data package are used to answer the research 
questions . 

I Demographic Data 

Demographic data were collected using the “TPCT Trainee Demographic Data” sheet 
from the TPCT training data package (see Appendix A). In the onginally proposed 
methodology, the demographic information was intended to provide data for the development 
of a model that would predict performance in the simulator. The proposed model was 
patterned after a model based on demographic factors that Cymrot (1990) concluded were 
linked to performance outcomes of naval officers who qualified as Surface Warfare Officers 


(SWOs). The qualification process includes tasks that resemble some of the tasking items in 
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the TPCT such as decision making during shipboard maneuvering. Included in Cymrot’s 
study were commissioning source, education, and shipboard experience. As discussed earlier 
in this thesis, the planned analysis of performance was abandoned due to inability of the 
participants to complete the scenario. 

az. TPCT Pre-Training Information 

The pre-training information consists of two questions developed for this research to 
examine prior experience with training simulators (see Appendix B). Prior experience was 
examined to check for possible bias in the sample group. It was first hypothesized that among 
SWOs there would not be very much simulator experience and that positive experience with 
simulators would bias the trainee’s impression of the TPCT toward more favorable responses. 
The first question asks if the trainee has ever used an interactive training simulator. Question 
two is only answered if the trainee responds positively to question one. Question two asks 
the trainee to rate the prior experience on a scale of one to five, with negative and positive 
as the anchor points, and ineffective or effective as the anchor points for the second part of 
the question. The last part of question two asks trainees for a “yes” or “no” answer on 
whether they felt training simulators were practical devices for future use in the Navy. 

3: Post-Training Questions 

A set of nine questions was developed to be administered orally after completion of the 
training (see Appendix C). The first five questions from this set were administered, then the 
trainee was given a battery of survey questions to complete (which is discussed next), then 
the trainee was asked the remaining four questions from this set. The first five questions ask 


the trainee specific questions about pressure (not stress) felt during the simulation, inadequacy 


in the student guide or instructor’s explanations. ~rceived learning, and perceived benifits 
from the training experience. The first three qu...ons of the remaining four ask specific 
questions regarding levels of stress, causes of stress, and attitude toward the stress caused by 
the trainer. The last question solicits an unstructured response by requesting 
recommendations and additional comments. 

4. NASA Task Load Index 

The “Task Load Index” (““TLX’’) was developed at NASA (Hart & Staveland, 1988) 
to assess the relative importance of six factors in determining how much workload" was 
experienced during a task or exercise. The “NASA-TLX” was modified from its original 
form and used to evaluate the task loading of trainees in this simulation (see Appendix D). 
The original scale is a ten-centimeter-long line, anchored at the low end at 0 and at the high 
end at one-hundred millimeters, scored by measuring the subject’s mark on the line to the 
nearest millimeter Professor Baumann, University of Illinois, who used the “TLX” to 
evaluate the Integrated Damage Control Training Technology (IDCTT), recommended 
changing the scale because “measuring things to the nearest millimeter seems like over-kill 
on a self report measure and struck us as likely to just introduce more random fluctuation” 
(1996b). The scale was modified to a 10-point scale anchored at the low end at zero and at 
the high end at 10. The modified “NASA-TLX” contains five two-part questions and one 


single-part question. The questions are divided into the following task loading factors: 1) 


'™"Workload” and “task loading” are generic terms that describe the pressures, 
effort, and stresses, both mental and physical, that occur when a person is involved in a 
task. 
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Mental Demand, 2) Physical Demand, 3) Temporal Demand, 4) Performance Demand, 5) 
Effort Demand, and 6) Frustration. The TPCT was designed to induce stress during the 
decision making process and these factors are used to measure how well the system 
accomplishes this goal. With the exception of “physical demand,” a measure of effectiveness 
for the system is scores above the midpoint of “5" on the scale. The criterion for 
effectiveness is arbitrary based on scores above 5 indicating that the demand was perceived 
as high rather than low. 

, User Interface Dimensions 

The “User Interface Dimensions” developed by Reeves and Harmon (1994) was used 
as modified by Johnson (1994) for use in evaluation of the IDCTT and further modified for 
use in this research (see Appendix E). This research used the eight dimensions as modified 
by Johnson and further changed the scale of one to 11, to zero to 10 for ease of analysis. The 
eight dimensions are: 1) Ease of Use, 2) Navigation, 3) Cognitive Load, 4) Mapping, 5) 
Knowledge Space Compatibility, 6) Information Presentation, 7) Media Integration, and 8) 
Overall Functionality. Each dimension is addressed as a separate question designed to 
provide a measure of the adequacy of the human-system interface. A low score for any one 
dimension can provide answers to where the system may need specific improvement. Scores 
above “5” on the scale indicate an appropriate interface. The criterion was established based 
on the original scoring that indicated marks to the nght of center were positive. 

6. TPCT Post-Training Information Survey 

The post-training information survey was developed to provide additional input for 


answering the research questions in this thesis (see Appendix F). This survey contains 16 
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questions that ask the trainee to rate the system’s ease of use, the scenario’s complexity, the 
comfort level of the environment, usefulness of the system as a trainer, effectiveness of some 
elements, problems encountered, satisfaction, and recommendations. Although they were 
developed using Johnson’s (1994) “STUDENT IDCTT SURVEY” as guidance, all of the 
questions in this survey are specific to the TPCT system and its scenario. Each question 
either provides specific input into areas of the trainer that may need improvement or provides 
an additional measure of effectiveness based on trainee satisfaction. 
C. PROCEDURE 

An oral background introduction to the system was conducted by the SIR instructor 
and each trainee was informed of the thesis research involved. As part of this introduction, 
the following statement was read to each trainee: 

During this training period you will be assisting in the beta test process of the 

TPCT. Your couperation in completing pre- and post-training surveys will be 

greatly appreciated. The information gathered in the surveys is for research 

purposes only, to assist in the installation of the TPCT. In no way will your 

inputs be utilized for other than research. Your candid comments are welcomed 

and encouraged. Your performance is being examined for the purpose of 

evaluating the TPCT system. 
Pnior to testing, each trainee was asked to complete the demographic and pre-test survey 
portion of the TPCT training data package in accordance with the facilitator’s guidelines 
section of the training data package (see Appendix G). 

After the introduction, the researcher acted solely as an observer, taking notes on 
comments and questions from the trainee, physical actions of the trainee, interactions between 


the trainee and the instructor, and responses by the instructor to the trainee’s questions. Each 


trainee was allotted a 15-30 minute period to review the student guide and ask questions. A 
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complete introduction was then given to the student by the instructor that covered operating 
the system, operating the IBS, and what role the instructor would play. 

The instructor’s role changed after two trainees when it was realized that more 
guidance was needed to understand key operating features of the system. The most prevalent 
change was the inclusion of a more detailed brief on the IBS and instructor assistance in this 
area. From this pomt on, the instructor operated the IBS for the student if necessary. Ifthe 
trainee made a decision that required the use of the IBS, recognized that IBS was needed, and 
did not know how to operate the IBS to implement the decision, the instructor could be asked 
for assistance. The instructor would only guide the trainee on the operating procedure 
required to implement the decision, not in the decision making process. The trainee could ask 
questions, but the instructor refrained from answering them as much as possible to maintain 
the autonomous decision making situation. 

Each trainee participated in the scenario until a decision point was missed, and the 
system indicated an error. The instructor would then debrief the mission to the point of the 
error, explaining and discussing the decision and training points to this pomt in time. The 
trainee could also ask system operation questions and comment freely on the system. After 
completion of the debrief, the scenario was restarted from the beginning, and the trainee 
would again proceed until a decision point was missed. Each trainee was usually only able 
to participate in the scenario twice, at the most making it half-way through the scenario on 
the second try, during the allotted 3-hour training session. 

After completion of the debrief for the last scenario try, the trainee was asked to 


complete the post-training evaluation. The post-training phase consisted of five structured 


interview questions followed by completion of the “Task Load Index,” “User Interface 
Dimensions”, and TPCT Post-Training Information survey portion of the TPCT training data 
package. The researcher was available to answer questions concerning the survey during this 
time. After completing the survey, the trainee was then interviewed with four more questions, 
the last being an open-ended request for recommendations and comments. 

At the completion of each training session, the researcher and the instructor discussed 
the session and recommendations that were made. The TPCT also generated a report for 
each session on the actions taken by trainee. The report indicated Decision Points reached 
and actions completed by the trainee. The results of the observation, survey results, 
interviews, and scenario reports provide the data for the formative evaluation of the TPCT. 


The formative evaluation of the data are presented and discussed in the next chapter. 
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V. DATA PRESENTATION AND ANALYSIS 


This chapter presents the data obtained using the Tactical Patrol Craft Trainer (TPCT) 
training data package,” the researcher’s observations, and the trainees’ comments. The data 
are presented as frequency distributions for the quantitative elements of the Pre-Training 
survey, “NASA Task Load Index” (“TLX’’), “User Interface Dimensions,” and TPCT Post- 
Training Information survey. Additionally the statistical mean and standard deviation are 
presented as appropriate. 

Because of their qualitative nature, the observations and comments are summarized 
based on significance and frequency. The significance of the observations or comments are 
judged subjectively by the researcher and influenced by the research questions outlined in this 
thesis. Frequency is not a criteria for inclusion, but it does contribute to significance. 
Comments from trainees are summarized. 

A. TPCT PRE-TRAINING INFORMATION 

The Pre-Training Information survey (see Appendix B) was developed to determine 
the computer-based simulator experience level of the trainees and examine any preconceived 
opinions about this technology prior to introduction to the TPCT. Question #1 asked the 
trainees if they had ever used an interactive training simulator (yes or no). Ten of 18 (56 
percent) of the trainees indicated that they had prior simulator experience. The trainees that 


answered yes to question #1 were asked to answer the three parts of question #2. Question 


'*The contents of this package are detailed in Chapter IV and included in this thesis 
as Appendices A thru G. 
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#2A asked the trainees to rate their previous experience as negative or positive. These ten 
trainees rated their experience as positive with a mean score of 4.5 with a standard deviation 
of 0.71 ona scale of 1 to 5. Question #2B asked the trainees to rate the previous training as 
ineffective or effective. These ten trainees rated the training as effective with a mean score 
of 4.1 with a standard deviation of 0.57 ona scale of 1 to 5. One-hundred percent of the 
group with prior experience also answered “yes” to question #2C, indicating that they 
envisioned future applications for computer-based simulator training in the Navy. The 
number of trainees in the TPCT test group with prior simulator experience was higher than 
expected, and the experiences were positive. This finding may bias the results of the research 
because trainees who enjoy the type of training they are involved in are more likely to respond 
favorably to questions about the effectiveness of the training (Ellis, 1986). 

B. TASK LOAD INDEX EVALUATION 

The ““TLX” developed by NASA and modified for this research is divided into six 
factors (see Appendix D). These factors are used in evaluating the TPCT’s ability to impose 
high levels of various cognitive demands that contribute to stress during the decision making 
process. The data for each factor are presented and discussed separately. 

Mental Demand experienced by the trainees is evaluated using data from the two parts 
of question #1 as displayed in Figure 5. The first part (A) identifies how much mental demand 
was required. The mean score was 7.78 with a standard deviation of 1.35. The second part 
(B) identifies the degree of difficulty in mental demand. The mean score was 7.11 with a 
standard deviation of 1.32. The trainees responded that the task required high levels of mental 


activity and was demanding. 
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Task Load Index : Mental Demand 
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Figure 5. Task Load Index questions 1A and 1B percentages. 
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Physical Demand experienced by the trainees is evaluated using data from the two 
parts of question #2 as displayed in Figure 6. The first part (A) identifies how much physical 
activity was required. The mean score was 1.28 with a standard deviation of 1.23. The 


second part (B) identifies the difficulty of physical activity. The mean score was 3.89 with 
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Figure 6. Task Load Index questions 2A and 2B percentages. 
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a standard deviation of 1.75. The trainees respond that the task required very little physical 
activity. But, even though the data show the physical activity was not difficult, it should be 
noted that over 40 percent of the respondents scored the difficulty at or above the set 
criterion of 5.'° This rating may be caused by an observed variance in the trainee’s ability to 
easily operate the touchscreen monitor (the only physical aspect of the trainer). 

Temporal Demand experienced by the trainees is evaluated using data from the two 
parts of question #3 as displayed in Figure 7. The first part (A) identifies how much time 
pressure the trainee felt due to the pace of the task. The mean score was 6.22 with a standard 


deviation of 2.05. The second part (B) identifies whether the trainee viewed the pace of the 


Task Load Index: Temporal Demand 
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Figure 7. Task Load Index questions 3A and 3B percentages. 
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'S The criterion for evaluating survey instruments scaled 0 to 10 was established in 
Chapter III. Low and high are determined relative to the midpoint of 5 on the scale. 
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task as slow or rapid. The mean score was 6.28 with a standard deviation of 1.27. Although 
both parts indicate high temporal demand, variation in the amount of time pressure felt is 
distinctive because the TPCT scenario was designed to create a highly time-sensitive 
environment. The data show that the pace is rapid with most responses occurring at or above 
a score of 6, but the resulting pressure experienced by the trainees varied considerably with 
over 30 percent responding at or below the criterion of 5. 

Performance Demand experienced by the trainees is evaluated using data from the two 
parts of question #4 as displayed in Figure 8. The first part (A) indicates how successful the 
trainee felt in the accomplishment of the tasks. The mean score was 6.28 with a standard 
deviation of 2.47. The second part (B) identifies how satisfied the trainee was with individual 


performance. The mean score was 4.67 with a standard deviation of 2.68. The data indicate 


Task Load Index: Performance Demand 
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Figure 8. Task boat Index questions 4A and 4B percentages. 
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that the trainees generally felt more successful than they felt satisfied. However, there are 
large variances in the responses to both questions. The inability, due to time restrictions, to 
complete the scenario may have impacted this factor. The trainees may have felt some 
success from advancing beyond some difficult task in the scenario, yet satisfaction could be 
diminished because the trainee could not complete the entire scenario. 

Effort Demand experienced by the trainees is evaluated using data from the two parts 
of question #5 as displayed in Figure 9. The first part (A) identifies how much mental effort 
was required to achieve the trainee’s level of performance. The mean score was 6.5 witha 
standard deviation of 1.34. The second part (B) identifies how much physical effort was 
required to achieve the trainee’s level of performance. The mean score was 1.78 with a 


standard deviation of 2.26. The data indicate that on average, the mental effort required to 


Task Load Index: Effort Demand 
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Figure 9. Task Load Index questions 5A and 5B percentages. 
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accomplish a level of performance was significantly higher than the physical effort. But notice 
should also be given to the several outliers who appeared to have some physical difficulties 
in performance achievement. The mental effort is expected in a computer-based simulator, 
but physical effort was not expected to be a factor. Difficulty with the touchscreen was 
reported by some trainees and may have contributed to this finding. 

Frustration experienced by the trainees is evaluated using data from question #6 as 
illustrated in Figure 10. The data indicate levels of frustration the trainees felt during the 
simulation. The mean score was 5.3 with a standard deviation of 1.94. Although the mean 
score is above the criterion of 5, the majority of responses are grouped around this criteria 
indicating that the frustration levels were not extremely high. Some trainees expressed 
specific frustration at not being able to finish the scenario. Other factors that could contribute 
to frustration are lack of experience with the Integrated Bridge System (IBS) or problems 


interfacing with the system. 
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Figure 10. Task Load Index question #6 response percentages. 
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The cumulative results of the “TLX” indicate that the TPCT induces high mental 
demand and requires low levels of physical activity. The temporal demand factor is rated 
high, but the time pressure aspect is lower than expected. Frustration levels are within a 
reasonable range and should not be considered detrimental to the training process. The TPCT 
system appears to be effective in generating the intended stress levels of trainees. 

Co USER INTERFACE DIMENSIONS 

The “User Interface Dimensions” (see Appendix E) are used in this research to 
determine a measure of the adequacy of the human-system interface. Each scale is anchored 
by a negative evaluation of the interface (e.g., “Difficult to Use”) at the low end, and a 
positive evaluation at the high end of the scale (e.g., “Easy to Use”). The mean score of 


trainees’ responses to all eight dimensions are shown in Figure 11. 
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Figure 11. “User Interface Dimensions” results. 
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All dimensions of the TPCT were evaluated above the criterion of “5" for 
consideration as an adequate human-system interface. The “Ease of Use” dimension had a 
mean score of 6.94 with a standard deviation of 1.35. The “‘Navigation’” dimension had a 
mean score of 6.61 with a standard deviation of 2.06. The “Cognitive Load” dimension had 
a mean score of 7.67 with a standard deviation of 1.61. The “Mapping” dimension had a 
mean score of 7.06 with a standard deviation of 2.07. The “Knowledge Space Compatibility” 
dimension had a mean score of 7.50 with a standard deviation of 1.50. The “Information 
Presentation” dimension had a mean score of 7.61 with a standard deviation of 1.72. The 
“Media Integration” dimension had a mean score of 7.61 with a standard deviation of 1.04. 
The “Overall Functionality” dimension had a mean score of 8.44 with a standard deviation 
of 1.15. Dimensions 1, 3, and 5 thru 8 were all positive with no indications of failure in the 
human-system interface, but the data for dimensions 2 (Navigation) and 4 (Mapping) 
indicated possible problems. 

Although “Navigation” was not below the set criterion, it was lower than the rest of 
the dimensions. Over 20 percent responded at or below the criterion of 5, with two trainees 
responding as low as 3 on the scale. The trainees expressed concern that they were unfamiliar 
with the system and had to find things “on the fly.” Suggestions were made to allow the 
trainee more time to review the student guide and allow the main screen to be accessible prior 
to starting the scenario for the trainee to get familiar with the buttons, their functions and 
contents. The low score in the “Navigation” dimension relative to other dimensions supports 
recommendations made by trainees to provide the trainees more familiarization with the 


interface before use. 
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Although the “Mapping” dimension’s mean was well above the criterion of 5, over 
40 percent of the trainees responded at or below the criterion. This variation in the dimension 
may be explained in two different ways. Two trainees asked for clarification of the dimension 
while taking the survey. This dimension was also questioned during the survey review 
conducted by Naval Postgraduate students. Confusion on this question may have added to 
its variation, but the fifth dimension was also questioned and required clanfication to some 
of the trainees and it did not have the same large variation. Some trainees expressed a feeling 
of being “lost in the scenario,” of “not knowing that I was behind schedule.” The ability to 
follow the progress and keep to a rigid time line is a major training point in the system and 
some trainees may have evaluated the “Mapping” dimension somewhat low because they felt 
there should be more clues when they were falling behind. But the trainees also expressed 
that this was not a bad aspect of the trainer because it reinforced ideas of “keeping an eye on 
the big picture.” 

Even with possible improvements indicated by the “Mapping” and “Navigation” 
dimensions, the interface was perceived as appropriate. The cumulative evaluation of the 
TPCT, using the “User Interface Dimensions,” is that it is easy to use and the trainees felt 
the media integration and overall functionality were good. 

D. TPCT POST-TRAINING INFORMATION SURVEY 

The post-training information survey (see Appendix F) results are consistent with the 
previous survey instruments and provide additional input for answering the research questions 
in this thesis. Each question is discussed in the following paragraphs. 


Question 1 asks for the trainee’s perception of the TPCT system’s ease-of-use. The 
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average score was 3.9 with a standard deviation of 0.9 ona 1 to 5 scale. The results indicate 
that the majority of trainees perceived the trainer as easy to use. This question specifically 
validated the results of the “User Interface Dimensions” ease-of-use dimension and indicated 
that the system, as a whole, was easy to use and thus the interface was not a major problem. 

Question 2 looked more specifically at the touchscreen interface to rate the trainee’s 
perception of its ease-of-use. This was examined because Johnson (1994) found that the 
IDCTT touchscreen had been a problem. The average score for the TPCT was 4.17 with a 
standard deviation of 0.7 ona 1 to 5 scale. The results indicate that the trainees considered 
the touchscreen easy to use. Although earlier results indicate possible problems with the 
touchscreen, the results here indicate that the difficulties experienced by the trainees were not 
severe. 

Question 3 was intended to evaluate the trainee’s perception of the scenario instead 


of the system or interface. The results of this question’s four parts are shown in Table 2. 


Table 2. Post-training information survey question 3 results. 


TPCT SCENARIO 


SCALE AVERAGE 


Easy to Difficult 3.4 
Confusing to Understandable 4.0 
Slow to Fast So 
Unrealistic to Realistic or 


‘Scale: lee S | 
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The scenario was designed to be understandable yet difficult, fast, and realistic. The average 
score for each part to this question was above the midpoint of 3 and indicates that the 
scenario was understandable while remaining difficult, fast, and realistic. 

Question 4 looks at environmental factors that may have contributed to the trainee’s 
perception of the trainer. Because the trainer was located in a temporary area that may have 
proved distracting, the trainees were asked to rate the temperature, comfort level, and noise 
level. The trainees rated the temperature neither too cold nor too hot with an average score 
of 3.0 with a standard deviation of 0.34 on a scale of 1 to 5. Comfort level was scored 
toward “Very Comfortable” by the trainees with an average score of 3.83 with a standard 
deviation of 0.62 ona scale of 1 to 5. The noise level did not seem to disturb the trainees 
who indicated that it was neither too quiet nor too loud with an average score of 3.0 witha 
standard deviation of 0.34 on a scale of 1 to 5. So, the trainees felt comfortable and the 
environment should not have affected their perceptions of the system. 

The TPCT system was designed as a trainer for the prospective PC class Commanding 
Officers and the trainees were asked in question 5 about the utility of the system to this end. 
Did the trainees think that the system was an effective trainer? The response to this question 
was very positive with an average score of 4.72 with a 0.46 standard deviation on a scale of 
1 to 5. The trainees also responded favorably to the question (number 6), which asked if 
there was utility for this type of system in other decision making training. The response to 
question 6 was an average score of 4.56 with a 0.70 standard deviation on a scale of 1 to 5. 
From these two questions, it appears that the trainees perceived the training as constructive 


despite any specific problems with the system. 
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Question 7 looks at some of the specific problems indicated in the literature review 
that may cause the system to appear ineffective to the trainees. The numbers of trainees that 
indicated difficulty with a specific aspect of the system are displayed in Table 3. Although the 
trainees indicated in other survey questions that the touchscreen was easy to use, five trainees 


expressed some difficulty with the technology. The most common comment was that the 


Table 3. Post-training information survey question 7 results. 


TRAINEE DIFFICULTIES 
Number of trainees who experienced difficulty with: TOTAL PERCENT 


Inputting information into the touchscreen monitor o 28 
Understanding audio reports 3 17 


Speed or volume of the information presented 0 


inputting information into the Integrated Bridge System (IBS) 39 


Other* 


ji Answers to "Other’ are explained in the text. 





trainee was not familiar with touchscreens and it “took some getting used to.” Three trainees 
indicated that they had difficulty understanding audio reports, but the cause was explained as 
“being involved in something else” or “not paying attention.” None of the trainees had 
difficulty with the speed or volume of information presented. The most common difficulty, 
as expected from the pre-testing, was inputting information into the Integrated Bndge System 


(IBS). Seven trainees expressed that they were not familiar with the IBS and this made the 
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entire training more difficult. Other difficulties expressed included “visualizing what was 
going on outside the ship” and “dealing with the system warping when I wasn’t ready.””* 

Question 8 asks the trainees how effective the student guide was in preparing them 
for the training. The response to this question was positive with an average score of 3.89 
with a standard deviation of 0.58 on a scale of 1 to 5. This is high considering there were 
numerous suggestions made to allow the trainee more time to study the student guide. One 
tramee was asked about this contradiction and commented that the guide was good, but more 
time was needed to digest the information. Although more time may be needed with the 
student guide prior to using the system, the student guide appears to be an appropniate pre- 
training instrument. Suggestions for improvements to the student guide are discussed later 
in this chapter. 

Questions 9 and 10 rate the trainee’s perception of instructor effectiveness and 
amount of information provided during the scenario. The trainees appeared to be very 
satisfied with the information provided by the instructor prior to operating the system. 
Question 9 had an average score of 4.44 with a standard deviation of 0.51 ona scale of 1 to 
5. Instructor intervention did not seem to disturb the trainees who indicated in response to 
question 10 that the instructor provided neither not enough nor too much information with 
an average score of 3.22 with a standard deviation of 0.55 on a scale of 1 to 5. So, the 
trainees felt comfortable with the information provided by the instructor prior to and during 


the scenario. 


“The simulation is shortened through the use of “warping,” which advances the 
trainee and ship’s position in the mission at an accelerated rate. During this time no 
actions are taken by the trainee. 
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Question 11 rates the trainees’ feelings of success during the scenario. The 
expectation was that either the trainees would not feel successful because they did not 
complete the scenario, or they might feel successful based on what they may have learned. 
The responses to this question indicate that the trainees did not feel very successful with an 
average score of 2.44 with a 1.10 standard deviation on a scale of 1 to 5. Individual 
comments from trainees support the assertion that the data representing perception of success 
are correlated with completion of the scenario. 

Question 12 asked for input on problems encountered while using the TPCT. The 


following list represents a summary of the trainees’ specific problems: 


u Warping compounded mistakes and created confusion. 

° Trainees could not “ask” the computer for recommendations. 
° They were unfamiliar with the touchscreen. 

2 They were unfamiliar with the IBS. 

° The IBS voyage planner stopped operating properly. 


The problem with the IBS voyage planner was software related and corrected immediately. 
The perceived difficulty with the touchscreen resulted in a recommendation for the addition 
of a “demo button” on the trainer screen to allow practice on the user interface without 
activating the scenario. The difficulty with the IBS and the perceived need to verbally 
interface with the system are discussed in further detail later in this chapter. The warp 
function was determined to be correctable if the trainee is indoctrinated into the procedure 
for making adjustments while warping. The system will stop warping if the trainee initiates 


an action, 1.e., orders a course or speed change. 
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Question 13 asked for input on what aspects of the TPCT the trainees liked the most. 
The following list represents a summary of the trainees’ responses: 
° Realism 


© Great graphics 


6 Detail 

° Audio reports 
° Time pressure 
e Touchscreen 


Ten trainees responded that they really liked the realism of the system. The graphics, detail, 
and audio are integral to the high fidelity system and essential to the perception of reality. 
The pressure of time management was liked by the trainees even though this was determined 
by the “TLX” to be a stressor. The trainees indicated that the time pressure was a realistic 
aspect of ship board operations. 

Question 14 asked for input on what aspects of the TPCT the trainees liked the least. 


The following list represents a summary of the trainees’ responses: 


0 Blank video screen, lack of video at times 

° Dead time 

° Glare on the trainer monitor 

° Not being able to ask watchstanders questions 
° Uncontrollable warping 


The most common dislike was the absence of video when not receiving reports from the 


system. The trainees indicated that this detracted from the reality and recommended a 
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continuos video stream. Additionally, the trainees indicated that video could be used to 
provide visual cues to aid in the decision making process, i.e., video of a fog bank may 
influence the decision to slow the ship or not. The trainees did not like the “dead time” 
between events. This appeared to cause anticipation stress, and it was not determined if this 
stress was beneficial or a hindrance to the goal of the system. The glare on the trainer 
monitor was noted by several trainees, and it was recommended that the trainer monitor be 
changed to an etched finished monitor similar to the IBS monitor. The verbal interaction with 
watchstanders 1s discussed later in this chapter. 

Question 15 asked for specific recommendations for future use of the TPCT. The 


following list represents a summary of the trainees’ recommendations: 


0 Training for all PC-bound officers 

° Ship handling scenarios, 1.e., anchoring evolutions, refueling, and man 
overboard 

° More mission-related scenarios 

° Casualty control scenarios 

° Watch team training and training for other watch stations 


Question 16 asked for specific recommendations for future use of the simulation 
technology used in the TPCT. The following list represents a summary of the trainees’ 
recommendations: 

° Engineering Casualty Control (ECC) drills 

° Damage control training 

° Similar training for different ship types 


The post-training information survey data indicate that the TPCT is easy to use, liked 
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by the trainees, and realistic. But, the survey also indicated some improvement areas and 
resulted in several recommendations. The survey is further supported by information gained 
through observation and trainees” comments. 

E. OBSERVATIONS AND ADDITIONAL TRAINEE COMMENTS 

During each training session the trainees were observed and notes were taken on their 
comments and questions, physical actions of the trainee, interactions between the trainee and 
the instructor, and responses by the instructor to the trainee’s questions. This section 
describes those observations as they apply to the research questions and compare to other 
data instruments. 

The “TLX” data show that the TPCT induces stress as intended. Observation of the 
trainees also indicated that stress was created by the system. The trainees would fidget, wnng 
their hands, and hesitate to make some decisions. Although the trainees would not admit to 
being stressed, they did indicate that they felt “pressure” to make the nght decisions and 
maintain the time line. They also indicated pressure as a result of the fast pace as well as 
unfamiliarity with various components. 

At the start of each training session, the touchscreen monitor appeared to create stress 
in the trainees. But the trainees were observed becoming more comfortable with the 
technology by the start of the second scenario run. Most trainees even seemed to enjoy the 
use of the touchscreen. This observation combined with the data from the post-training 
survey supports a recommendation made by the trainees to alter the system to allow practice 
on the touchscreen to build familiarity. Recommendations were also made to investigate 


voice recognition as an alternate means of interfacing with the system. 
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The trainees were observed “talking” to the system. Most often, the trainees would 
say “no” out loud to deny a recommended action. If the system recommended a course 
change and the trainee did not agree, then no action on the part of the trainee was required 
to deny the request, but the trainees expressed a need to actually do something either verbally 
or physically to indicate a “no” response. 

The trainees were also observed to constantly ask for information from the instructor 
to assist them in the decision making process. As an example, one trainee asked if the ship 
had departed enemy waters yet. This is information that the trainee should be able to find or 
would have known if attention was paid to audio reports. Ifthe instructor answered these 
questions, the scenario would become too easy. The instructor for this evaluation usually 
responded to these questions with “I can not give you that information.” After discussing this 
with the instructor, it was decided that to simulate reality maybe the instructor could answer 
as a crew member might with something along the lines of “I’ll get back to you on that, sir.” 
The instructor recommended clarifying this in the instructor’s guide. But, it should be noted 
that the data show that the trainees were happy with the information provided. 

The instructor’s guide also did not include specific information on conducting the pre- 
training brief. Although the instructor for this evaluation conducted excellent briefs and the 
trainees were satisfied with the information presented, discussions with the instructor resulted 
in a recommendation that the brief be standardized and included in the instructor’s guide. 

The pre-training brief for this evaluation included an extensive review of the IBS that 
could almost be considered training in itself. The IBS is a complicated navigation and control 


system that requires some detailed instruction and practice to master. Even with this bnef, 
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the trainees experienced problems operating the IBS mostly because they were unfamiliar with 
the system. The trainees recommended being taught the IBS before using the TPCT. 

The trainees noted the following items that detracted from the reality of the system. 
At one point in the scenario the fuel percentage indicated 2 percent fuel remaining. It was 
noted by the trainees that the PC class ship will lose fuel suction and fail to operate if the 
percentage drops below 10 percent, so 2 percent and still operating is not realistic. In the 
event of loss of steering gear, the PC class ship has the capability of “steering with engines” 
while this is not an option in the TPCT. The TPCT has static gauges that do not change in 
the case of an engine casualty. It was recommended by trainees that the gauges be 
programmed to indicate actual casualty conditions. Reality was also questioned when it was 
observed that the successful completion of a decision point only required a minimal decrease 
in speed instead of a decrease in speed according to standard ship board procedure. 

The trainees kept asking for a device known as a “whiz wheel.” The small plastic 
hand-held device is used on ships to calculate time-distance problems. Although the time- 
distance problems can be calculated by hand, it was recommended that a “whiz wheel” be 
made available for the trainees to use. Additionally, the trainees did not easily understand 
how to utilize the IBS system in conjunction with the clock to ensure they were on schedule. 

The trainees appeared somewhat agitated that they could not finish the scenario in the 
3-hour time frame. The instructor and trainees recommended that the scenario include an 
alternate starting point to reduce the time involved in starting over. But it was also noted that 
the trainees expressed a desire to return and try the system again, especially if more and varied 


scenarios were added. 
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The observation of training resulted in qualitative support for earlier findings that the 
TPCT induces stress as intended, the trainees like the system, it is relatively easy to use, and 
trainees perceive the training as beneficial. This phase of the evaluation also resulted in 
recommendations for improvements to the system. The following chapter summarizes the 


findings and presents recommendations based on the data. 
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VI. CONCLUSIONS AND RECOMMENDATIONS 


The purpose of this chapter is to present the results of the formative evaluation of the 
Tactical Patrol Craft Trainer (TPCT) with regard to the research questions presented in the 
introduction. The literature review indicated that formative evaluation is important to the 
development of a successful training system, and prior research provided insight on specific 
evaluation tools to use and an indication of expected results. The methodology used in this 
research was directed at answering specific research question, the results of which are 
presented in the following sections. 

A, RESEARCH QUESTIONS 

This research was based on the four questions presented in chapter 1 of this thesis. 
The last two questions in the introduction were eliminated from this study for reasons 
discussed in the methodology chapter and are not discussed here. The first question 
contained five parts that asked about specific system changes that were needed for the trainer. 

1. What Changes Need to Be Made in the Instructor’s Guide? 

The instructor’s guide provides a clear and concise description of the TPCT system, 
but it needs to include some specific direction for the instructor. The guide needs to include 
a section that details a standard pre-training brief to ensure continuity among instructors and 
among trainees. Included in this brief, the instructor should provide the trainee with 
instruction on using the Integrated Bridge System (IBS) in conjunction with the trainer clock 
to solve time management problems. The brief should also contain clarification on responding 


to a trainee’s request for information during the scenario. 
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Ze What Changes Need to Be Made in the Student’s Guide? 

The Student’s Guide was found to be adequate if the trainee is provided with 
sufficient time to study it. It is recommended that the student be provided with instruction 
in the guide on using the trainer clock with the IBS to solve time management problems. 

3. What Software Problems Need to Be Corrected? 

The software operates as designed and the one noted programming problem was 
corrected. However, various recommendations that involve software changes are listed as 
follows: 

0 Add an alternate starting point. This change would allow the trainee to begin 

a second or third session closer to where there was a missed decision point, 
reducing the training time, making it possible to complete the scenario, and 
aiding the perception of success. 

' Add a “Permission Denied” button. Although this button would not affect the 
scenario, it would give the trainees the desired physical action to deny a 
request from the system instead of simply doing nothing. 

> Add a “demo button” to allow familiarity with the user interface. 

' Add steering with engines capability. This added feature will add another 
element of realism, allowing the trainee to take an action that is available on 


the PC class ship. 


° Adjust fuel to indicate a minimum of 10 percent because the current setting 
of 2 percent is not realistic. 


° Change required reduction in speed during fog. The speed reduction required 
by the trainee should be in accordance with standard operating procedures 
instead of an arbitrary amount. 


° Add temperature changes during engine casualties. 
° Add continuous video feed and visual cues. 
* Add additional scenarios. 
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4. What Hardware Problems Need to Be Corrected? 

There were no major problems noted with the TPCT hardware. However the 
recommendation 1s made to change the trainer monitor to a monitor with an etched finish to 
reduce glare. Additionally, the hardware needs to be located in a dedicated training area that 
is free from distractions. 


5: What Additional Instruction Needs to Be Conducted Prior to Using the 
Trainer? 


The TPCT trainer relies heavily on the use of the IBS, a complicated system with 
which the trainees do not have familiarity. It is recommended that a prerequisite to TPCT 
training course of instruction be developed that teaches the IBS. The IBS component of the 
TPCT can be used to conduct this training. This training would also provide the trainee with 
valuable experience on a system that will be on that trainees ship. Other additional instruction 
items needed were noted in recommended improvements to the Instructor’s and Student’s 
guides. 


6. What Initial Indications of System Effectiveness are Produced During 
Beta Testing? 


No absolute measures of effectiveness were produced by this research. The research, 
through the use of survey instruments and trainee perceptions, generated the following 
indications that the TPCT operated as expected or was perceived as effective. 


° The TPCT induces high mental demand. 


° The system appears to generate the intended stress levels in trainees. 
° The human-system interface is adequate and appropriate. 
° The system is relatively easy to use. 


. Trainees liked the system. 


. Trainees felt the system was realistic. 
° Trainees perceived the training as effective. 
° Trainees perceived the system as useful. 


B. ADDITIONAL FINDINGS AND RECOMMENDATIONS 

This research resulted in additional findings and recommendations that were not 
specific to the original research questions. The following list summarizes these findings and 
recommendations: 

° Perceived performance and success in the training do not appear to affect 
perceptions of training utility. Although the trainees did not consider 
themselves successful in the scenario, they still rated the TPCT as a useful 
training device. 

° Most prior experience with computer-based training was positive. 

° It is recommended that the system be adapted to train other watchstanders 
because other watchstanders can encounter the same decision making 


problems that the TPCT is designed to enhance. 


» It is recommend that the TPCT system be used for team training, especially 
bridge watchteams that are involved in assisting the CO in decision making. 


° It is recommend that the technology be used to develop trainers for 
Engineering Casualty Control (ECC), ship handling, and other ship types. 


° It is recommended that the use of voice recognition be investigated as an 
alternative user interface. 


C. RECOMMENDATION STATUS 
During the evaluation of the TPCT, recommendations were submitted to Commander 
Naval Special Warfare Command (COMNAVSPECWARCOM) for review. The following 


recommendations were approved for implementation: 
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° Add an alternate starting point. 


. Add a “Permission Denied” button. 

. Add a “demo button” to allow familiarity with the user interface. 
. Add steering with engines capability. 

» Adjust fuel to indicate a minimum of 10 percent. 

: Change required reduction in speed during fog. 

> Add temperature changes during engine casualties. 


D. RECOMMENDATIONS FOR FURTHER RESEARCH 

Two aspects of further research must be considered. First, any changes to the system, 
especially software additions, need to undergo continued formative evaluation to ensure the 
system works properly. Formative evaluation similar to that conducted in this thesis is also 
recommended for new scenarios or applications of this technology to other areas or ship 
types. It is also recommended that further research be conducted on the TPCT after 
implementation to determine training effectiveness. 

This research produced initial indicators of effectiveness. Summative evaluation of 
the TPCT system needs to be conducted to check actual effectiveness. Outcome-based 
measures of effectiveness need to be developed and tested on trainees that complete a course 
of instruction on the TPCT. Additionally, trainees can be surveyed at a later time to gain 


information on perceived utility of the system. 
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APPENDIX A. TRAINEE DEMOGRAPHIC SURVEY 


Mark all that apply (if you had two jobs as a division officer, mark them both). 


Last Name: Ship Types Served in 
CRUDES O 
First Name: AMPHIB O 
CV ) 
SSN: Auxiliary O 
PC ) 
Age: MCM/MHC a) 
Other: L) 
Rank: (specify 
YG: Assignment history 
Command at sea Q) 
XO UW 
Education Background Department Head 
BS engineering W OPS QO 
BS non-engineering OQ) Deck QO 
BA degree Q) Nae QO 
MS engineering, ) Eng QO 
MS non-engineering Q) CSO QO 
MA degree Q) Weps QO 
aa : Other: Q) 
er: 
Te (specify) 
Division Officer 
Commissioning Source Eng a) 
USNA Lu Deck O 
NROTC OQ) Ne QO 
OCS a) Ops QO 
ECP ) Weps L) 
Seaman to Admiral ) cS QO 
Other: J Other: a) 


(specify) 


(specify) 








APPENDIX B. PRE-TRAINING INFORMATION SURVEY 


1. Have you ever used an interactive training simulator? 
Yes U No U 

2. Only answer the following if #1 was “Yes”. 
A. Rate your experience: 


Negative Positive 
] ------------------ 2------------------ 3 ------------------ 4------------------ 5 


B. Was the training: 
Ineffective Effective 


[eseceeeranee oe ae peers ease ee (aee sesh ee 5 


C. From your experience, could you see future uses for that type of training in the 
Navy? 


Yes U No U 
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APPENDIX C. POST TRAINING ORAL QUESTIONS 


Ask the following questions immediately after completion of the last scenario run for the 
student: 

Record answers in the space below. 

1. Did you feel pressured to complete a specific task during the mission? 


2. Where were you the most pressured? 


3. Was there anything you needed to know pnior to starting that was not in the Student 
Guide or explained by the instructor? (What?) 


4. Do you feel as if you have learned anything new?(What?) 


5. What other benefits did you experience from using the TPCT? 


Administer the Post-test surveys (Sources-of-Workload Evaluation, User Interface 
Dimensions, and the TPCT Post-Training Information survey) , then ask questions 6 thru 
Sh 

6. Did any aspect of the trainer cause you to feel stressed? 

7. What aspect caused you to feel stressed? 


8. Do you consider this stress a positive or a negative aspect of the trainer? 


9. Is there anything you would like to add, recommendations or comments? 


7) 





APPENDIX D. NASA TASK LOAD INDEX 


The evaluation you are about to perform is a technique that has been developed by NASA 
to assess the relative tmportance of six factors in determining how much workload you 
experienced. The procedure is simple: Read the following task descriptions and then mark the scale 
at the point that reflects the task load that you experienced after completing the trainer. If you have 
any questions, please ask them now. Thank you for you participation. 


1. MENTAL DEMAND 


A. How much mental and perceptual activity was required (e.g., thinking, deciding, calculating, 
remembering, looking searching, etc.)? 


Low High 
0 1 2 3 4 5 6 a 8 869 10 


B. Was the task easy or demanding, sumple or complex, forgiving or exacting? 
Low High 
0 1 2 = 3 4 5 6 © 8 9 10 
2. PHYSICAL DEMAND 


A. How much physical activity was required (e.g., pushing, pulling, turing, controlling, 
activating, etc.)? 


Low High 
lil 2 3 a 5 ofl 8 9 10 


B. Was the task easy or demanding, slow or brisk, slack or strenuous, restful or laborious? 


Low High 
0 1 2 3 4 5 6 7 8 9 10 


3. TEMPORAL DEMAND 


A. How much time pressure did you feel due to the rate or pace at which the tasks or task elements 
occurred? 


Low High 
Ome 2 «3 4 5 6 i 9 10 


B. Was the pace slow and leisurely or rapid and frantic? 


Low High 


4. PERFORMANCE DEMAND 


A. How successful do you think you were in accomplishing the goals of the task set by the 
experimenter (or yourself)? 


Low High 
RS ee) ee 4 5 6 7 8 9 10 


B. How satisfied were you with your performance in accomplishing these goals? 


Low High 
0 1 2 3 4 5 6 7 8 9 10 


5. EFFORT DEMAND 
A. How hard did you have to work mentally to accomplish your level of performance? 


Low High 
0 12 3 4 5 6 7 8 9 10 


B. How hard did you have to work physically to accomplish your level of performance? 


Low High 
OT. 2 3 4 5 6 7 8 9 10 


6. FRUSTRATION 


A. How insecure, discouraged, irritated, stressed and annoyed versus secure, gratified, content, 
relaxed and complacent did you feel during the task? 


Low High 
0 12 3 4 5 6 7 8 9 10 
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APPENDIX E. USER INTERFACE DIMENSIONS 


Directions: A number of statements that describe the Interactive-multimedia Courseware (ICW) 
are given below. Read each statement and then circle the number that reflects your opinion. There 
are no right or wrong answers. 


Dimension I - Ease of Use 
(Perceived facility with which user interacts with the ICW) 


Difficult Easy 
0 1 2 3 4 5 6 7 8 9 10 


Dimension 2 - Navigation 
(Pperceived ability to move through the contents of the ICW) 


Difficult Easy 
0 1 2 3 4 5 6 7 8 9 10 
Dimension 3 - Cognitive Load 


(Perceived degree that the user interface seems manageable) 


Unmanageable Manageable 
0 1 2 = 3 4 5 6 7 es 9 10 


Dimension 4 - Mapping 
(Program's ability to track and graphically represent user's path through the program) 


None Powerful 
0 1 2 3 4 5 6 7 8 9 10 


Dimension 5 - Knowledge Space Compatibility 


(Concepts and relationships were compatible with the user's knowledge about the topic) 


Incompatible Compatible 
OCs? 3 4 5 6 7 8 9 10 
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Dimension 6 - Information Presentation 
(Perceived degree that the information contained in the ICW is presented in an understandable 
form) 


Obtuse Clear 
0 1 2 3 4 5 6 | 8 9 10 


Dimension 7 - Media Integration 
(How much does the ICW coordinate the different media to produce an effective whole) 


Uncoordinated Coordinated 
0 1 2 3 4 5 6 7 8 9 10 


Dimension 8 - Overall Functionality 
(Perceived utility of the ICW in relation to the program’s intended use) 


Dysfunctional Highly Functional 
OF Aly 2 3 4 5 ome 7 8 9 10 
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APPENDIX F. POST-TRAINING INFORMATION SURVEY 


1. Rate how difficult or easy the TPCT system was to operate. 


Difficult Easy 

] ------------------ 2------------------ 3 ------------------ 4------------------ 5 
2. Rate how difficult or easy the touch screen was to operate. 

Difficult Easy 

| ------------------ 2------------------ 3 ------------------ 4.----------------- 5 


3. Rate the TPCT Scenario according to the following criteria (note: This question refers 
to the battle problem itself and not the TPCT system). 


Easy Difficult 
] ------------------ 2------------------ 3 ------------------ 4------------------ 5 
Confusing Understandable 
] ------------------ 2------------------ 3------------------ 4------------------ 5 
Slow Fast 
] ------------------ 2------------------ 3------------------ 4------------------ 5 
Unrealistic Realistic 
| ------------------ 2------------------ 3 ------------------ 4------------------ 5 
4. Rate the environment according to the following criteria. 
Too Too 
Cold Hot 
| ------------------ 2------------------ 3 ------------------ 4------------------ 5 
Very Very 
Uncomfortable Comfortable 
] ------------------ 2------------------ 3 ------------------ 4------------------ 5 
Too Too 
Quiet Loud 
| - eee eas... 3 eee aan fn ta oa 5 
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5. Rate the usefulness of the TPCT as a trainer for prospective PC class Commanding 
Officers. 


Completely Very 
Worthless Useful 
‘es a eee ae wenn eer 5 


6. Rate the usefulness of this type of simulation for other decision making training 


Completely Very 
Worthless Useful 
]------------------ 2---+--------------- oe ee 5 


7. Mark any area that caused you difficulty while using the TPCT. 
QO) Inputting information with the touch screen monitor 
L) Understanding audio reports 


L) Speed or volume of the information presented (did you easily lose track of the 
situation due to the speed or volume of the information flow?) 


) Inputting information into the Integrated Bridge System (IBS) 
LY Other (please specify): 


A. In the space provided below, explain why the items you checked caused you 
difficulty. 


8. Rate the effectiveness of the Student Guide (Did it prepare you for operation of the 
TPCT?) 


Ineffective Effective 
a Dios eee 3. av {a 5 


9. Rate the effectiveness of the instructor in providing needed information prior to 
operation of the TPCT system. 


Ineffective Effective 
ee secon ses ee a 5 
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10. Did the instructor provide too much or not enough information during the scenario? 


Not Enough Too Much 
] ------------------ 2------------------ 3 ------------------ 4.----------------- 5 


11. How successful do you feel you were during the training scenano? 


Not Successful Very Successful 
] ------------------ 2------------------ 3 ------------------ 4.----------------- 5 


12. What problems did you encounter while using the TPCT? 


13. What aspects of the TPCT did you like the most? 


14. What aspects of the TPCT did you like the least? 


!5. What recommendations would you make for future use of the TPCT? 


16. What recommendations would you make for future use of similar trainers? 


We 





APPENDIX G. FACILITATOR’S GUIDELINES 


Directions for conduct of training for data gathering purposes. The following is a 
guideline that will assist the facilitator in completing all necessary information for research 
purposes. 


1. Have student read Student Guide, provide 15-30 minutes if needed, note the amount of 
time the student actually reads the guide and record any questions asked. 


2. Brief student on system IAW instructor’s guide and include the following: 
“During this training period you will be assisting in the beta test process of the 
TPCT. Your cooperation in completing pre and post training surveys will be 
greatly appreciated. The information gathered in the surveys is for research 
purposes only, to assist in the installation of the TPCT. In no way will your inputs 
be utilized for other than research. Your candid comments are welcomed and 
encouraged. Your performance is being examined for the purpose of evaluating 
the TPCT system.” 
Have student fill out demographic survey. 
Record any questions asked by the student during the brief. 

3. Start mission. 
Record questions, statements, and significant actions (i.e., Student looks confused, 
student yells “NO” to OOD, student says “What do I do now?”, even if no answer 
Is given). 


Record any assistance given to the student. 


4. At end of mission: Debrief the mission, explaining missed training points, answer 
student questions, and record these items. 


5. Run mission again until success or time expiration, continue to evaluate the training 
using numbers 3 and 4 above until completion. 


6. Administer post-training interview and surveys after the final debref. 


7. Collect all surveys, notes and mission evaluation sheets into one package for each 
student. 


8. For the facilitator: note any problems encountered in administering these data sheets: 
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